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DETAILED ACTION 



1 . Claims 2-41 are in this application, claims 24-26 and 36 are canceled by the 
examiner amendment, claims 2-15, 20, 22-23, 28-35, 37 and 39-42 are pending in this 
application. 



2. An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1 .312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 



Authorization for this examiner's amendment was given in a telephone interview 
with Emmanuel Rivera on 05/1 3/04. 

Please cancel these claims 24-26 and 36. 



initiating a search for images based on at least one query keyword in a query; 
identifying, during the search, first images having associated keywords that match the 
query keyword, and second images that contain low-level features similar to those of 
the first images identified as matching the query keyword, wherein the low-levels 
features comprise color, shape, and texture and the low-level features do not match the 
search query; 

ranking the first and second images; 



EXAMINER'S AMENDMENT 
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presenting the first and second images to a user; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image.-- 



^ please replacjlclaini/l^ith;-- 4j A method comprising: 

initiating a search for images based on at least one query keyword in a query; 
identifying, during the search, first images having associated keywords that 
match the query keyword, and second images that contain low-level features similar to 
those of the first images identified as matching the query keyword wherein the low- 
levels features comprise color, shape and texture and the low-level features do not 
match the search query; 

presenting the first and second images to a user; 

monitoring feedback from the user as to which of the first and second images are 
relevant to the query; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image.- 
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j-^^j /please replace|claim^^ith;~ A method comprising: 

initiating a search for images based on at least one query keyword in a query; 
identifying, during the search, first images having associated keywords that 
match the query keyword, and second images that contain low-level features similar to 
those of the first images identified as matching the query keyword wherein the low- 
levels features comprise color, shape, and size and the low-level features do not match 
the search query; 

presenting the first and second images to a user; 

receiving feedback from the user as to whether the first and second images are 
relevant to the query; 

learning how the first and second images are identified based on the feedback 
from the user; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image. 




£^PIease replac^claim Jj^j^ith;- 1 A method comprising: 

permitting entry of both keyword-based queries and content-based queries; 
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finding a first set of images using semantic-based image retrieval and a second set of 
images using low-level feature-based image retrieval based on the first set of images 
wherein the low-levels features comprise color, shape and texture and the low-level 
features do not match the semantic-based and the low-level features do not match the 
keyword-based or content-based; 

presenting the images to a user so that the user can indicate whether the images 
are relevant; 

conducting semantic-based relevance feedback and low-level feature-based 
relevance feedback in an integrated fashion; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the images 
deemed relevant by the user and/or assigning a small weight to an association between 
the query keyword and the example image.-- 



Please cancel claim 24. 
Please cancel claim 25. 
Please cancel claim 26. 



Please replace/claimJoKvith;-- 20J A method comprising: 
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identifying the set of images from the presenting retrieved using the at least one 
l<eyword and second images that contain low-level features similar to those of the first 
images identified as matching the at least one keyword, wherein the low-levels features 
comprise color, shape and texture and the low-level features do not match the search 
query; 

presenting a result set of images that are returned from an image retrieval search 
of a query having at least one keyword; 

monitoring feedback from a user as to whether the images in the result set are 
relevant to the query; 

in an event that the user selects at least one image as being relevant to the 
query, associating the keyword in the query with the selected image to fonri a first 
keyword-image association and assigning a comparatively large weight to the first 
keyword-image association; and 

in an event that the user identifies an example image for refinement of the 
search, associating the keyword in the query with the example image to form a second 
keyword-image association and assigning a comparatively small weight to the second 
keyword-image association. ~ 



/^lease repla^ claim ^^ith;~ 28^ An image retrieval system comprising: 
a query handler to handle both keyword-based queries having one or more 
search keywords and content-based queries having one or more low-level features of 
an image; and 
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a feature and semantic matcher to identify at least one of (1 ) first images having 
keywords that match the search keywords from a keyword-based query, and (2) second 
images having low-level features similar to the low-level features of a content-based 
query based on the first images, wherein the feature and semantic matcher ranks the 
images, and the low-level features comprise color, shape and texture, and the low-level 
features do not match the keyword-based or content-based; 

presenting the first and second images to a user; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image. 



please replacejclaim ^^th;~ 32^ An image retrieval system comprising: 
a query handler to handle both keyword-based queries having one or more 
search keywords and content-based queries having one or more low-level features of 
an image; 

a feature and semantic matcher to identify at least one of (1 ) first images having 
keywords that match the search keywords from a keyword-based query, and (2) second 
images having low-level features similar to the low-level features of a content-based 
query based on the first images, wherein the low-level features comprise color, shape 
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and texture and the low-level features do not match the keyword-based or the content- 
based; 



a user interface to present the images identified by the feature and semantic 
matcher to a user, the user interface allowing the user to indicate whether the images 
are relevant to the query; and 

a feedback analyzer to train the image retrieval system based on user feedback 
as to relevancy, wherein the feedback analyzer to assign a large weight to an 
association between the query keyword and the images deemed relevant by the user 
and/or feedback analyzer to assign a small weight to an association between the query 
keyword and the example image and/or a feedback analyzer to assign a small weight 
to an association between the search keywords and the example im age.-- ■ 



computer-executable instructions that, when executed, directs a computer to: 
find a first set of images using both semantic-based image retrieval, and a second set of 
images using low-level feature-based image retrieval based on the first set of images, 
wherein the low-level features comprise color, shape and texture and the low-level 
features do not match the sematic-based or feature-based; 

present the images to a user so that the user can indicate whether the images 
are relevant; and 



r 



Please cancel claim 36. 
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concurrently conduct semantic-based relevance feedback and low-level feature- 
based relevance feedback, 

wherein the receiving feedback from the user as to whether the first and second 
images are relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the images 
deemed relevant by the user and/or assigning a small weight to an association between 
the query keyword and the example image.-- 



Allowable Subject Matter 

3. Claims 2-1 5, 20, 22-23, 28-35. 37 and 39-42 are allowed over prior art made of 
record. 

The following is an examiner's statement of reasons for allowance: None of the non 
prior art alone or incombination neither teaches nor suggest "initiating a search for 
images based on at least one query keyword in a query; identifying, during the search, 
first images having associated keywords that match the query keyword, and second 
images that contain low-level features similar to those of the first images identified as 
matching the query keyword, wherein the low-levels features comprise color, shape, 
and texture and the low-level features do not match the search query; ranking the first 
and second images; presenting the first and second images to a user; and receiving 
feedback from the user as to whether the first and second images are relevant to the 
query further comprising: assigning a large weight to an association between the query 
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keyword and the images deemed relevant by the user and/or assigning a small weight 
to an association between the query keyword and the example image." 

Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 

Conclusion 

4. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

AgganA/al et al. (US. Patent No. 6,728,706 62) Patent date: 06/03/2003 
Foxetal. (US. Patent No. 6,574,632 62) Patent date: 03/03/2003 

Shevade et al., "an experiental annotation system". Year 2003, IS6A: 1-5081 13, 
page 91-98. 

Lai et al., "P6IR-MM: multimodal image retrieval and annotation", Year 2002, 
IS5N: 1-58113-620-X, page 421-422. 

Contact information 

5. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to 6aoquoc N. To whose telephone number is (703) 305- 
1949 or via e-mail 6aoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM -4:30 PM, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John 6reene can be reached at (703) 305-9790. 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 
Commissioner of Patents and Trademarks 
Washington, D.C. 20231. 

The fax numbers for the organization where this application or proceeding is 
assigned are as follow: 

(703) 872-9306 [Official Communication] 

Hand-delivered responses should be brought to: 
Crystal Park II 
2121 Crystal Drive 
Arlington, VA 22202 
Fourth Floor (Receptionist). 

Baoquoc N. To /I 



May 13, 2004 




